Latency reduction in feedback-based system performance determination

ABSTRACT

The present disclosure is directed to a technique to reduce latency in feedback-based system performance determination. A system receives, from an application developer device, indications of an in-application event and a first input value for an application content delivery profile. The system receives, via an interface from an application developed by an application developer and executed by a computing device remote from the data processing system and different from the application developer device, a ping indicative of an occurrence of the in-application event on the computing device. The system merges data from the ping with internal data determined by the data processing system to generate merged data. The system determines a predicted performance for the in-application event and provides an indication of the predicted performance. The system configures, responsive to the indication of the predicted performance, the application content delivery profile with a second input value.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application claims the benefit of priority under 35 U.S.C. § 120 asa continuation of U.S. patent application Ser. No. 15/196,891, filedJun. 29, 2016, which is hereby incorporated by reference herein in itsentirety.

BACKGROUND

A graphical environment, applications or web pages can include text,images, video, or audio information provided via an application serveror web page server for display. Additional content item objects can alsobe provided by third parties for display on the application or the webpages together with the information provided by the entities. Thecontent item objects can include instructions to cause installation ofan application configured for execution on a computing device. Thus, aperson viewing a graphical environment can access the information thatis the subject of the web page, the selected third party content itemobjects that may or may not be related to the subject matter of the webpage, and trigger the instructions embedded in the content item objectto cause installation of the corresponding application. Due to the largenumber of available content item objects and the delay in determiningperformance associated with an installed application, it may bechallenging to select and provide content item objects for display inthe graphical environment.

SUMMARY

At least one aspect is directed to a system to reduce latency infeedback-based system performance determination. The system can includea data processing system comprising one or more processors and memory.The data processing system can receive, from an application developerdevice, one or more indications of an in-application event and a firstinput value for an application content delivery profile. The dataprocessing system can receive, via an interface from an applicationdeveloped by an application developer and executed by a computing deviceremote from the data processing system and different from theapplication developer device, a ping indicative of an occurrence of thein-application event on the computing device. The data processing systemcan merge data from the ping with internal data determined by the dataprocessing system to generate a merged data set. The data processingsystem can determine, based on the merged data set and the first inputvalue, a predicted performance for the in-application event. The dataprocessing system can provide, via the interface, an indication of thepredicted performance determined based on the merged data set and thefirst input value. The data processing system can configure, responsiveto the indication of the predicted performance, the application contentdelivery profile with a second input value.

At least one aspect is directed to a method of reducing latency infeedback-based system performance determination. The method can includea data processing system receiving, from an application developerdevice, one or more indications of an in-application event and a firstinput value for an application content delivery profile. The method caninclude the data processing system receiving, via an interface from anapplication developed by an application developer and executed by acomputing device remote from the data processing system and differentfrom the application developer device, a ping indicative of anoccurrence of the in-application event on the computing device. Themethod can include the data processing system merging data from the pingwith internal data determined by the data processing system to generatea merged data set. The method can include the data processing systemdetermining, based on the merged data set and the first input value, apredicted performance for the in-application event. The method caninclude the data processing system providing, via the interface, anindication of the predicted performance determined based on the mergeddata set and the first input value. The method can include the dataprocessing system configuring, responsive to the indication of thepredicted performance, the application content delivery profile with asecond input value.

BRIEF DESCRIPTION OF THE DRAWINGS

The details of one or more implementations of the subject matterdescribed in this specification are set forth in the accompanyingdrawings and the description below. Other features, aspects, andadvantages of the subject matter will become apparent from thedescription, the drawings, and the claims.

FIG. 1 is an illustration of a system to reduce latency infeedback-based system performance determination in accordance with animplementation.

FIG. 2 is a flow diagram illustrating operation of a system to reducelatency in feedback-based system performance determination in accordancewith an implementation.

FIG. 3 is a graphical user interface of a system to reduce latency infeedback-based system performance determination in accordance with animplementation.

FIG. 4 is an illustration of a method for reducing latency infeedback-based system performance determination in accordance with animplementation.

FIG. 5 is a block diagram illustrating a general architecture for acomputer system that may be employed to implement various elements ofthe systems shown in FIGS. 1 and 2, the graphical user interface shownin FIG. 3, and the method shown in FIG. 4 in accordance with animplementation.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

The present disclosure is generally directed to reducing latency infeedback-based system performance determination. For example, whenexecuting a content campaign for a mobile application, it may bechallenging for a content provider or application developer to determineor estimate an expected performance for a content item object comprisinginstructions to download and install the application, or perform aspecific action within the application. Predicting performance can beincreasingly challenging as content item objects for the application arerendered on various web properties, applications, or networks.Furthermore, it is challenging to predict different types of events suchas future application installations versus in-application events becausethese different events may utilize different tracking interfaces. Thus,it may be challenging for a content provider or application developer toconfigure a content campaign (e.g., application content deliveryprofile) to cause increased installations of their application due tothe delay in obtaining feedback on the performance of the contentcampaign, application, or system.

Systems and methods of the present disclosure can reduce latency infeedback-based system performance determination. For example, a dataprocessing system can leverage an improved tracking interface and datamerging techniques along with statistical modeling techniques to providepredicted performance data based on input values.

The data processing system can provide guidance regarding input valuesfor a content campaign comprising an application content deliveryprofile, such as bid and budget guidance, by providing a real-timeestimate of performance. For example, a content provider can leverage anestimate of application installations determined by the data processingsystem or volume and revenue of in-application events selected by thecontent provider. The data processing system can provide this estimatevia a graphical user interface, and further update the estimate inreal-time while the content provider creates or modifies their contentcampaign. The projected estimates can adapt to a variety of inputs. Forexample, inputs can include, but are not limited to, bids (willingnessto pay for an application installation), budgets (daily or total spendlimit), select countries (geographic location of computing devices thatrender the ad), languages (of the device settings which will render thead), or type of content item (e.g., whether the content provider ispromoting a specific video, image). By allowing various inputs andsurfacing projections in real-time, the data processing system allowscontent providers to make media buying decisions in real-time, andunderstand the tradeoffs of their various choices.

To generate these projections, the data processing system can usemachine learning and multivariable regression techniques. The dataprocessing system can use a combination of data internal the dataprocessing system and data provided by an application developer orcontent provider. The internal data can include, for example, datarelated to organic application impressions (e.g., impressions not inresponse to an ad), clicks, selections, installations, or spend, as wellas ad logs of impressions, clicks, or conversions. Data provided by acontent provider or application developer can include, for example, avariety of in-application events and values for those in-applicationevents. The data processing system can use a model that combines ormerges the internal data and the data provided by the content providerto predict performance for new and existing applications.

For example, for existing applications, the data processing system canuse both the content history and interactions associated with thatapplication to predict the likelihood that other (similar) computingdevices install that application or complete the in-application action.The data processing system can determine, based on logged contenthistory, computing devices or computing device profiles that are likelyto click, install and engage with content item objects for theapplication.

For content providers of new applications that have previously listedtheir application in an online marketplace, the data processing systemcan leverage organic data to predict which computing devices are likelyto click on a listing for the application, likely to install theapplication, and likely to engage with or otherwise perform an actionsvia the application.

For applications that are new to the online marketplace that have notpreviously listed in an application in the online marketplace, the dataprocessing system can determine a characteristic of the application toidentify similar applications, and use information about a category,geography, and language of the identified similar applications todetermine a likely performance.

Thus, the data processing system provides a robust, real-time, accurateinterface that can combine organic internal data and data from pingsfrom the application and uses a machine learning technique to predictperformance of application installations or an in-application eventbased on one or more input values. Input values can include, forexample, assigning an in-application event a certain value (e.g.,booking a flight from New York City to Paris can have a value of $500value for a flight), or the in-application event can be assigneddifferent values or a range value for each computing device, or instanceof the action. The data processing system can tailor the predictions tothe specific content provider, application, settings, and goals inreal-time and reduce the delay in feedback-based system performance. Thedata processing system can provide tradeoffs in real-time and changesettings to optimize content delivery. The data processing system canlink input to expected output for mobile application install ads orre-engagement ads.

For example, the data processing system can obtain historical data forcontent item objects that are configured to cause an application to bedownloaded or installed on a computing device (e.g., an ad with a linkto the application on an online marketplace). The historical data caninclude one or more signals, such as impression (e.g., the content itemobject rendered on a computing device), click (e.g., a user of thecomputing device selecting the content item object rendered on thecomputing device), installation (e.g., the user of the computing deviceinstructing the computing device to download and install the applicationon the computing device), or conversion (e.g., a user performing adesired in-application event via the installed application executing onthe computing device). The data processing system can attribute theinstallation of the application (or in-application action) to therendering of the content item data object on the computing device. Forexample, the data processing system can correlate time stamps associatedwith the content item impression and installation, device identifiers,profile information, location identifiers, language identifiers,application category information, publisher information, developerinformation, tags, or other indications or identifiers.

The data processing system can aggregate or merge the signals across allusers of the same application to create a first merged data set. Thedata processing system can aggregate or merge signals across allapplications that are similar to the application (e.g., all travelapplications) that were downloaded via one or more content item objectsto create a second merged data set. The data processing system canaggregate or merge the signals across all similar applications that wereorganically downloaded (e.g., were not installed responsive to a contentitem object or ad) to create a third merged data set. In some cases, thedata processing system can merge the first, second and third data sets.

Signals can include internal signals determined by the data processingsystem or signals provided by an interface of the application executingon the computing device. Internal signals can include, for example, typeof application, whether the application is installed, a configuration ofthe application, or whether payments have been made or paymentinformation is linked to the application. Signals provided by theapplication can include, for example, application statistics, state ofthe application, levels achieved in the applications, scores achieved inthe application, queries input into the applications, or transactionsconducted via the application. The application developer can configurethe in-application events to provide signals to the data processingsystem. The data processing system can store the data in a datastructure such as {content_item_ID, event_ID}.

The data processing system can be configured with a machine learningtechnique that takes as input the signals to predict which of thesignals are correlated to a desired event, such as conducting atransaction. The data processing system can determine a score for eachsignal. The data processing system can map the scored signals tocomputing devices in real-time. For example, the data processing systemcan identify computing devices that match the highest scoring signals todetermine a number of computing devices having the highest scoringsignals. Based on the highest scoring signals, the data processingsystem can predict a likelihood of whether the desired event (e.g.,application installation or in-application event) will occur for thecomputing device. Based on the number of computing devices having thehighest scoring signals, the data processing system can determine aperformance of the content campaign or application. The data processingsystem can provide the predicted performance to a content provider orapplication developer in real-time.

FIG. 1 illustrates an example system 100 to reduce latency infeedback-based system performance determination accordance with animplementation. The system 100 can include content selectioninfrastructure. The system 100 can include a data processing system 120communicating with one or more of a content provider computing device125, application developer device 125, content publisher computingdevice 115, online marketplace provider 175, or client computing device110 via a network 105. The network 105 can include computer networkssuch as the Internet, local, wide, metro, or other area networks,intranets, satellite networks, and other communication networks such asvoice or data mobile telephone networks. The network 105 can be used toaccess information resources such as mobile applications, web pages, websites, domain names, or uniform resource locators that can be displayedon at least one computing device 110, such as a laptop, desktop, tablet,personal digital assistant, smart phone, or portable computers. Forexample, via the network 105 a user of the computing device 110 canaccess mobile applications or web pages provided by at least one website operator or content publisher 115 or application developer 125. Aweb browser of the computing device 110 can access a web server of theweb site operator or content publisher 115 to retrieve a web page fordisplay on a monitor of the computing device 110. The content publisher115 can provide instructions to the application executing on thecomputing device 110 to render the content. The application developer125 or web site operator or content publisher 115 generally includes anentity that operates the mobile application or web page. The applicationdeveloper 125 or web site operator or content publisher 115 includes atleast one web page server that communicates with the network 105 to makethe web page available to the computing device 110. The data processingsystem 120 can provide instructions to the application executing on thecomputing device 110 to render content. The computing device 110 canaccess the online marketplace provided by the online marketplaceprovider 175 via network 105. The computing device 110 can download anapplication 180 via the online marketplace via network 105.

The network 105 may be any type or form of network and may include anyof the following: a point-to-point network, a broadcast network, a widearea network, a local area network, a telecommunications network, a datacommunication network, a computer network, an ATM (Asynchronous TransferMode) network, a SONET (Synchronous Optical Network) network, a SDH(Synchronous Digital Hierarchy) network, a wireless network and awireline network. The network 105 may include a wireless link, such asan infrared channel or satellite band. The topology of the network 105may include a bus, star, or ring network topology. The network mayinclude mobile telephone networks using any protocol or protocols usedto communicate among mobile devices, including advanced mobile phoneprotocol (“AMPS”), time division multiple access (“TDMA”), code-divisionmultiple access (“CDMA”), global system for mobile communication(“GSM”), general packet radio services (“GPRS”) or universal mobiletelecommunications system (“UMTS”). Different types of data may betransmitted via different protocols, or the same types of data may betransmitted via different protocols.

The system 100 can include at least one data processing system 120. Thedata processing system 120 can include at least one logic device such asa computing device having a processor to communicate via the network105, for example with the computing device 110, the applicationdeveloper 125 or web site operator or content publisher computing device115 (or content publisher 115), and at least one content providercomputing device 125 (or provider device 125 or content provider 125 orapplication developer 125). The data processing system 120 can includeat least one computation resource, server, processor or memory. Forexample, the data processing system 120 can include a plurality ofcomputation resources or servers located in at least one data center.The data processing system 120 can include multiple, logically-groupedservers and facilitate distributed computing techniques. The logicalgroup of servers may be referred to as a data center, server farm or amachine farm. The servers can also be geographically dispersed. A datacenter or machine farm may be administered as a single entity, or themachine farm can include a plurality of machine farms. The serverswithin each machine farm can be heterogeneous—one or more of the serversor machines can operate according to one or more type of operatingsystem platform.

Servers in the machine farm can be stored in high-density rack systems,along with associated storage systems, and located in an enterprise datacenter. For example, consolidating the servers in this way may improvesystem manageability, data security, the physical security of thesystem, and system performance by locating servers and high performancestorage systems on localized high performance networks. Centralizing theservers and storage systems and coupling them with advanced systemmanagement tools allows more efficient use of server resources.

The data processing system 120 can include a content placement systemhaving at least one computation resource or server. The data processingsystem 120 can include at least one interface component 130, at leastone attribution component 135, at least one landscape generatorcomponent 140, at least one content selector component 145, and at leastone data repository 150. The at least one data repository 150 caninclude or store, in one or more data structures or data bases, internaldata 155 corresponding to one or more applications, signal data 160,merged data 165, or content data 170. The internal data 155 can includedata determined by the data processing system associated with theapplication, such as a type of application, configuration of theapplication, geographic location of a computing device when theapplication or content item is provided, or language associated with theapplication or computing device. The signal data 160 can includein-application events such as performing an electronic transaction viathe application, a game level or score of the application, or queriesinput into the application. Merged data 165 can include merging internaldata 155 and signal data 160 for an application or category ofapplications. Content data 170 can include, for example, contentcampaign information, content groups, content selection criteria,content item objects or other information provided by a content provider125 or obtained or determined by the data processing system tofacilitate content selection. The interface component 130, attributioncomponent 135, landscape generator component 140, and content selectorcomponent 145 can each include at least one processing unit or otherlogic device such as programmable logic array engine, or moduleconfigured to communicate with the database repository or database 150.The interface component 130, attribution component 135, landscapegenerator component 140, and content selector component 145, and datarepository 150 can be separate components, a single component, or partof the data processing system 120. The system 100 and its components,such as a data processing system 120, may include hardware elements,such as one or more processors, logic devices, or circuits.

The data processing system 120 can obtain anonymous computer networkactivity information associated with a plurality of computing devices110. A user of a computing device 110 can affirmatively authorize thedata processing system 120 to obtain network activity informationcorresponding to the user's computing device 110. For example, the dataprocessing system 120 can prompt the user of the computing device 110for consent to obtain one or more types of network activity information.The identity of the user of the computing device 110 can remainanonymous and the computing device 110 may be associated with a uniqueidentifier (e.g., a unique identifier for the user or the computingdevice provided by the data processing system or a user of the computingdevice). The data processing system can associate each observation witha corresponding unique identifier.

For situations in which the systems discussed here collect personalinformation about users, or may make use of personal information, theusers may be provided with an opportunity to control whether programs orfeatures that may collect personal information (e.g., information abouta user's social network, social actions or activities, a user'spreferences, or a user's current location), or to control whether or howto receive content from the content server that may be more relevant tothe user. In addition, certain data may be treated in one or more waysbefore it is stored or used, so that certain information about the useris removed when generating parameters (e.g., demographic parameters).For example, a user's identity may be treated so that no identifyinginformation can be determined for the user, or a user's geographiclocation may be generalized where location information is obtained (suchas to a city, ZIP code, or state level), so that a particular locationof a user cannot be determined. Thus, the user may have control over howinformation is collected about the user and used by a content server.

The data processing system 120 can include an interface component 130designed, configured, constructed, or operational to receive data froman application executed on a computing device or provide a graphicaluser interface to a content provider 125. The interface component 130can execute on one or more processors or computation resources of thedata processing system 120 to receive data and provide information. Theinterface 130 can provide one or more widgets to receive input data,such as input values associated with a content campaign.

A content provider 125 may establish an electronic content campaign. Anelectronic content campaign can refer to one or more content groups thatcorrespond to a common theme. A content campaign can include ahierarchical data structure that includes content groups, content itemdata objects, and content selection criteria. To create a contentcampaign, content provider 125 can specify values for campaign levelparameters of the content campaign. The campaign level parameters caninclude, for example, a campaign name, a preferred content network forplacing content item objects, a budget for the content campaign, a bidamount for content item objects, start and end dates for the contentcampaign, a duration for the content campaign, a schedule for contentitem object placements, language, geographical locations, type ofcomputing devices on which to provide content item objects. Othercampaign level parameters can include, for example, a payment techniquesuch as a cost-per-click (CPC), cost per thousand impressions (CPM),cost-per-action (CPA), cost-per-conversion, or cost per thousandconversions. In some cases, an impression can refer to when a contentitem object is fetched from its source (e.g., data processing system 120or content provider 125), and is countable. In some cases, due to thepossibility of click fraud, robotic activity can be filtered andexcluded, as an impression. Thus, in some cases, an impression can referto a measurement of responses from a Web server to a page request from abrowser executed on the computing device 110, which is filtered fromrobotic activity and error codes, and is recorded at a point as close aspossible to opportunity to render the content item object for display onthe computing device 110. In some cases, an impression can refer to aviewable impression; e.g., the content item object is at least partially(e.g., 20%, 30%, 30%, 40%, 50%, 60%, 70%, or more) viewable on a displaydevice of the client computing device 110. A click or selection canrefer to a user interaction with the content item object, such as amouse-click, touch interaction, gesture, shake, audio interaction, orkeyboard click. A conversion can refer to a user taking a desired actionwith respect to the content item objection; e.g., purchasing a productor service, completing a survey, visiting a physical store correspondingto the content item, or completing an electronic transaction.

The content provider 125 can further establish one or more contentgroups for a content campaign. A content group includes one or morecontent item objects and corresponding content selection criteria, suchas keywords, words, terms, phrases, geographic locations, type ofcomputing device, mobile app category, publisher site, time of day,interest, topic, or vertical. Content groups under the same contentcampaign can share the same campaign level parameters, but may havetailored specifications for particular content group level parameters,such as keywords, negative keywords (e.g., that block placement of thecontent item in the presence of the negative keyword on main content),bids for keywords, or budget, geographies, languages, mobile appcategories, audiences, and publisher placements.

To create a new content group, the content provider can provide valuesfor the content group level parameters of the content group. The contentgroup level parameters include, for example, a content group name orcontent group theme, and bids for different content placementopportunities (e.g., automatic placement or managed placement) oroutcomes (e.g., clicks, impressions, or conversions). A content groupname or content group theme can be one or more terms that the contentprovider 125 can use to capture a topic or subject matter for whichcontent item objects of the content group is to be selected for display.For example, a developer of travel applications can create a differentcontent group for each type of travel application or application featureit develops. Examples of the content group themes that the travelapplication developer can use include, for example, “Flights”, “Hotels”,“Trains”, “Rental Cars”, “Cheap flights”, or “Beach Vacation”,“All-inclusive Resort Packages”. An example content campaign theme canbe “beach vacation” and include content groups for both “flights” and“hotels”, for example.

The application developer 125 or content provider 125 can provideoverall objectives or one or more keywords and content item objects toeach content group. Objectives can include “optimize volume of thesespecific in-app events” or “optimize profit among these specific in-appevents”. Keywords can include terms that are relevant to the product orservices of associated with or identified by the content item objects. Akeyword can include one or more terms or phrases. For example, thetravel application developer may include “beach,” “resort,” “family,”“tropical,” as keywords for a content group or content campaign. In somecases, negative keywords can be specified by the content provider toavoid, prevent, block, or disable content placement on certain searchresult webpages or content webpages matching those negative keywords.The content provider can specify a type of matching, such as exactmatch, phrase match, or broad match, used to select content itemobjects.

The content provider 125 can provide one or more content item objectsfor selection by the data processing system 120. The data processingsystem 120 (e.g., via content selector component 145) can select thecontent item objects when a content placement opportunity becomesavailable that matches the budget, content schedule, maximum bids,keywords, and other selection criteria specified for the content group.Different types of content item objects can be included in a contentgroup, such as a text content item, an image content item, video contentitem, audio content item, multimedia content item, call content item, orcontent item link. The content item object can include a link orinstruction that causes the computing device to download, install oraccess the application of the application developer 125. For example,the content item object can include a link to an online marketplaceprovider 175. The link can cause the computing device to access theapplication via the online marketplace 175, and download theapplication.

The data processing system 120 can include a content selector component145 designed, constructed, configured or operational to select contentitem objects. For example, to select content items for display in atextual environment, the data processing system 120 can parse the textto identify keywords, and use the keywords to select a matching contentitem based on a broad match, exact match, or phrase match. For example,the content selector component 145 can analyze, parse, or otherwiseprocess subject matter of candidate content items to determine whetherthe subject matter of the candidate content items correspond to thesubject matter of the textual information resource. The content selectorcomponent 145 may identify, analyze, or recognize terms, characters,text, symbols, or images of the candidate content items using an imageprocessing technique, character recognition technique, or databaselookup. The candidate content items may include metadata indicative ofthe subject matter of the candidate content items, in which case thecontent selector component 145 may process the metadata to determinewhether the subject matter of the candidate content item corresponds tothe web page or search query.

Content providers 125 may provide additional indicators when setting upa content campaign that includes content items. The content provider mayprovide information at the content campaign or content group level thatthe content selector component 145 may identify by performing a lookupusing information about the candidate content item. For example, thecandidate content item may include a unique identifier, which may map toa content group, content campaign, or content provider. The contentselector component 145 may determine, based on information stored incontent campaign data structure in data repository 150, informationabout the content provider 125.

The data processing system 120 can receive, via a computer network, arequest for content for presentation on a computing device 110. Therequest can include selection criteria of the request, such as thedevice type, location, and a keyword associated with the request. Thekeyword can be associated with the webpage on which the content is to bedisplayed, a search query input by the user of the computing device, orother information associated with the request or the computing device.

Responsive to the request, the data processing system 120 can select acontent item object from data repository 150 and provide the contentitem for rendering on the computing device 110 via network 105. Thecomputing device 110 can interact with the content item object. Thecomputing device 110 can select a hyperlink or other button in thecontent item object that causes or allows the computing device 110 toidentify an application, download the application, and install theapplication on the computing device 110.

The data processing system can include an attribution component 135designed, constructed and configured and operational to attribute anapplication installation or in-application event to a content itemimpression. The attribution component 135 can interact or communicatewith the content selector component 145 to identify a content itemimpression 145 that corresponds to the computing device 110 on whichapplication 180 is installed. For example, when the content selectorcomponent 145 provides the content item for display on the computingdevice 110, the content selector component 145 can store an impressionlog or content data 170. The content data 170 can include a record ofthe impression, such as an identifier of the content item, identifierassociated with the computing device, timestamp for the impression,geographic location of the computing device at the time of theimpression. The attribution component 135 can further identify thecomputing device that downloaded and installed the application (orotherwise accessed the application). For example, the data processingsystem 120 can include the online marketplace and provide theapplication to the computing device 110. If the data processing system120 provided both the content item and the application, the dataprocessing system 120 can use the same identifier for the computingdevice 110 or the data processing system 120 can maintain an index orhash function that maps a first identifier of the computing device usedto provide the content item to a second identifier of the computingdevice used to provide the application.

In some cases, the data processing system 120 can receive theapplication installation information from the online marketplaceprovider 175 or from the application developer 125. When the dataprocessing system 120 receive the application installation informationfrom another entity (e.g., the application developer 125 or the onlinemarketplace provider 175), the data processing system 120 can correlatethe installation with the content item. For example, the applicationinstallation information can include a time stamp for the request toinstall the application, metadata associated with the request to installthe application (e.g., source of the request to install, which caninclude a content item object or organic search for the application),geographic location of the computing device at the time of the request,or language. The data processing system 120 can use the time stamp ofthe request to download the application to identify content itemimpressions for the application that occurred within a predeterminedtime window of the request (e.g., within 1 minute, 5 minutes, 30minutes, 1 hour, 2 hours or more). In the event the data processingsystem 120 identifies multiple content item impressions, the dataprocessing system can further filter the content item impressions basedon geographic location of the computing device 110 at the time of thecontent item impression and request to download the application, andlanguage to identify the content item impression that corresponds to theapplication download. In some cases, the data processing system 120 canmap the identifier of the computing device to a common identificationscheme, or a common identifier can be used by both the data processingsystem 120 for content selection and application delivery.

Upon attributing the content item impression to the applicationdownload, the attribution component 135 can establish a computing deviceidentifier mapping. Thus, as subsequent in-application information isreceived from the computing device 110 for the application 180, the dataprocessing system 120 can attribute the information to the same contentitem impression in a data structure as follows: {application_ID,impression_ID, application_event_D}.

The data processing system 120 can include a landscape generatorcomponent 140 designed and constructed to receive signals and inputvalues and predict performances based on the received signals and inputvalues. The landscape generator component 140 can predict performancefor an application. Performance can refer to a number of installationsof the application attributed to a content item object for theapplication being rendered on computing devices 110. Performance canrefer to a desired in-application event that may or may not beattributed to the content item object.

The landscape generator component 140 can predict a number ofinstallations for the application. The number of installations can referto the number of different computing devices 110 that download theapplication and install the application. In some cases, the landscapegenerator component 140 can predict a number of downloads of theapplication and a number of installations of the application, where thenumber of downloads may be different from the number of installations.The number of downloads may be greater than the number of installationsif some of the computing devices 110 that downloaded the application didnot install the application, or were prevented from installing theapplication or were unable to install the application. For example, acomputing device 110 may receive an error when attempting to install theapplication if the computing device 110 is not properly configured toinstall the application or the user installing the application lacks therequisite authority to install the application or the computing device110 loses internet or wireless connectivity during the installationprocess or before the installation process can begin. In some cases, thenumber of installations may be greater than the number of downloads. Forexample, a single computing device may download the application from thedata processing system 120 or online marketplace 175, and may then sharethe downloaded file with other computing device 110 so the othercomputing devices 110 can install the application without having todownload the application from the data processing system 120 or onlinemarketplace 175.

The data processing system 120 can receive information used to predictperformance of a content campaign for an application. The contentprovider 125 can provide the data processing system 120 with anindication of a type of event. The data processing system 120 can thenpredict a number of occurrences of the event. For example, the dataprocessing system 120 can receive an indication of the following typesof events to optimize: application downloads, application installations,application setup, or an in-application event. The content provider 125or application developer 125 can provide the indication of this type ofevent during setup of the content campaign, or management of the contentcampaign.

The data processing system 120 can receive one or more input values. Theinput value can include a tuning parameter used by the content selectorcomponent 145 to select content items of the content provider 125 forrendering on computing devices 110. The tuning parameter can be used inan online auction. The online auction can include a real-time auction.Real-time auction can refer to an auction that runs after the dataprocessing system 120 receives a request for content for the computingdevice 110. The data processing system 120 can use the real-time auctionto identify one or more candidate content items. The data processingsystem 120 can select a candidate content item based on one or morescores generated in the auction, such as a relevancy score, ad score,quality score, or bid amount. For example, the data processing system120 can combine the bid amount with a quality score to generate an adscore, and select the content item object corresponding to the highestad score. Thus, the content provider 125 can increase or decrease thebid amount to increase or decrease the number of times the content itemobject of the content provider 125 is selected based on the results ofthe online auction.

The data processing system 120 can receive input values such as a bidamount, a max bid amount, min bid amount, budget, or return oninvestment for the content campaign. The data processing system 120 canuse the one or more input values to determine a performance. Forexample, if the bid amount is $2 and the budget is $100 per day, thedata processing system can determine that the content item objects ofthe content campaign might be selected 50 times before the budget runsout. The content provider 125 may choose to increase the budget or lowerthe max bid amount to increase the number of impressions.

To further predict performance, the data processing system 120 canreceive additional information. The data processing system 120 canreceive the information from a component of the data processing system120, online marketplace 175, or computing device 110. The dataprocessing system 120 can obtain internal data obtained or determined bythe data processing system 120. Internal data can refer to or includedata directly determined by the data processing system 120, or data thatthe data processing system 120 can access, or data stored in datarepository 150. For example, internal data can include a type ofapplication provided by the data processing system 120 to the computingdevice 110, language of the application, geographic location of thecomputing device 110 as determined by the data processing system usingreceived location information, a configuration of the application (e.g.,has the application been linked with a financial institution tofacilitate a financial transaction) or information associated with thecontent impression.

The data processing system 120 can receive data via signals or pingsfrom an application executed on computing devices 110. The dataprocessing system 120 can receive the signals via an interface from anapplication developed by the application developer 125. The dataprocessing system 120 can receive the information from a computingdevice 110 executing the application 180, where the computing device 110is remote from the data processing system 120 (e.g., communicates vianetwork 105 with data processing system 120). For example, the dataprocessing system 120 can receive a ping indicative of an occurrence ofthe in-application event on the computing device. The in-applicationevent can refer to a desired action occurring via the application, aquery, a level or score if the application is a type of game, atransaction or other action or event configured by the applicationdeveloper 125 to trigger a ping. The event can be associated with avalue; for example, booking a flight to Paris can correspond to a valueof $300, whereas booking a rental car via the application can correspondto a value of $50.

The data processing system 120 can provide a software development kit tothe application developer 125. The application developer 125 can use thesoftware development kit to configure the application to transmit pingsfor predetermined in-application events responsive to a trigger oroccurrence of the event. For example, the application 180 can transmit aping responsive to a user inputting a query for “flights to Paris”. Theapplication 180 can be configured to parse the query input into theapplication 180 (e.g., via an input text box, audio input, keyboard,mouse), identify that the query or terms thereof match a desired query,and transmit a ping indicating an occurrence of the in-application eventidentifier corresponding to the query.

The data processing system 120 can merge the internal data with the datafrom the pings to generate a merged data set. For example, the internaldata and ping data can include several signals. The data processingsystem 120 can attribute the signals to a content item impressionidentifier or computing device 110 identifier. The data processingsystem 120 can merge the internal data and ping data into a datastructure that relates to a common content impression identifier orcomputing device 110 identifier.

In some cases, the data processing system 120 receives multiple pingsfrom multiple applications executed by one or more computing devices.The data processing system 120 can categorize the pings based on thetype of applications or other characteristic related to theapplications. For example, the data processing system 120 can categorizethe applications into the following categories: entertainment, news,travel, games, productivity, utility, music, or video. The dataprocessing system 120 may further categorize or group applications basedon other characteristics, such as language, geography, or operatingsystem. The data processing system 120 can use the categories toidentify similar applications.

The data processing system 120 can merge data from the multiple pingswith internal data determined by the data processing system to generatethe merged data. The data processing system 120 can merge the multiplepings even though they may be for different applications because thedata processing system 120 can determine that the different applicationsare similar based on being in a same category or having one or morecharacteristics in common.

The data processing system 120 can determine, based on the merged dataset and the first input value, a predicted performance for thein-application event or a number of installations of the application.The data processing system 120 can determine the predicted performancebased on a machine learning technique. The machine learning techniquecan include, e.g., a regression model, a multivariate Poisson regressionmodel, or a multivariate Poisson regression model with regularizations.

The data processing system 120 can use the machine learning technique topredict which of the signals in the merged data are correlated to adesired event, such as an in-application event or applicationinstallation. The data processing system 120 can use the machinelearning technique to output a score for each of the signals or acombination of signals. The data processing system 120 can rank thesignals or combination of signals based on their scores to identify thetop ranking signals or combination of signals. For example, a topranking signal for a travel application where the desired event ispurchasing a flight ticket to Paris can be inputting a search query withterms “flights to Paris” or “flights to Paris this weekend” or “visitEiffel Tower” or “tourist attractions in Europe”.

The data processing system 120 can identify additional signals thatindicate a likelihood of occurrence of the desired event. For example,the data processing system 120 can determine, using the machine learningtechnique, that a signal corresponding to configuring the travelapplication card with a financial account (e.g., a credit card number,checking account, or other financial instrument) indicates a higherlikelihood of the user purchasing a flight to Paris, as compared totravel applications that are not configured with a financial account.Similarly, the data processing system 120 can determine that a user whopreviously purchased goods in a gaming application is likely to purchasegoods in gaming applications in the future.

The data processing system 120 can predict the performance of a contentcampaign for an application based on an input value using the mergeddata. In some cases, the data processing system 120 can predict theperformance of the content campaign for the application without usingping data from the application. For example, the data processing system120 can identify other applications that are similar to the applicationfor which the content campaign is being setup or managed, and predictthe performance of the application using ping data associated with thesimilar applications. Thus, the data processing system 120 can generatethe merged data with one or more of internal data, ping data of theapplication, or ping data of similar applications.

The data processing system 120 can use the machine learning technique togenerate scores for signals in the merged data. In some cases, the dataprocessing system 120 can use a Poisson regression with regularizations.A Poisson regression can refer to a regression analysis that modelscount data and contingency tables. When using the Poisson regression,the data processing system can determine that a response variable Y(e.g., a desired application event) has a Poisson distribution, andfurther determine that a logarithm of its expected value can be modeledby a linear combination of unknown parameters (e.g., signals of themerged data). In some cases, the data processing system 120 can use anegative binomial regression, which generalizes the Poisson regression.

In the Poisson regression, the dependent variable can be a count, suchas a number of application events. The events can be independent sincethe occurrence of one event on one computing device 110 is independent(e.g., does not make it more or less likely) from the occurrence of theevent on another computing device 110.

In some cases, the data processing system 120 can use a regularizedPoisson regression to identify a value for θ that maximizes thelikelihood of an expression of the form:

${\sum\limits_{i = 1}^{m}\; {\log \left( {p\left( {y_{i};e^{\theta^{\prime}\; x}} \right)} \right)}},$

where m is the number of examples in the merged data set, and p(y_(i);e^(θ′x)) is the probability mass function of the Poisson distributionwith the mean set to e^(θ′x). Regularization can be added to thisoptimization problem by maximizing:

${{\sum\limits_{i = 1}^{m}\; {\log \left( {p\left( {y_{i};e^{\theta^{\prime}\; x}} \right)} \right)}} - {\lambda {\theta }_{2}^{2}}},$

for some positive constant λ.

Upon identifying the signals correlated with the desired event, the dataprocessing system 120 can identify a number of computing devices 120 inreal-time that are associated with the top ranking signals. For example,the data processing system 120 can determine that the top rankingsignals for purchasing a flight to Paris are: search query for “flightto Paris” in application; application configured with credit card;application language is French; and the location of computing deviceexecuting application is Montreal, Canada. The data processing system120 can identify, in real-time, the number of computing devices 110 thatare associated with these signals. For example, the data processingsystem 120 can determine, based on ping data received from the computingdevices 110 and internal data stored in the data processing system 120related to these computing devices 110 that there are 1,000 computingdevices 110 matching these signals.

In some cases, the data processing system 120 can calibrate thepredicted performance based on a second predicted performance of aplurality of applications to determine the predicted performance. Theplurality of applications can be similar to the application based onbeing in the same category (e.g., travel applications), language,operating system, or geography. The data processing system 120 can use amerged data set to predict the performance. The data processing system120 can adjust the initial predicted performance based on a predictedperformance using the similar applications. For example, there may bemore historical data available for the similar applications if theapplication is a new application or relatively new application. Thus,the data processing system 120 can use the additional historical pingdata received for similar application to generate an improved or moreaccurate predicted performance, and provide the improved predictedperformance to the content provider 125.

The data processing system 120 can thus facilitate providing contentitems for the content provider 125 for rendering on computing devices110 having one or more of the top ranking signals. The content provider125 can adjust an input value, such as a bid amount or budget, andprovide an indication to deliver the content items to computing deviceshaving the one or more top ranking signals. The data processing system120 can provide an estimate or prediction of a number of applicationinstalls or in-application events that might result from the inputvalue. The content provider 125 may determine to adjust the input valueto increase or decrease the predicted number of applicationinstallations or in-application events. For example, the contentprovider 125 may determine that there are only 50 flights left fromMontreal to Paris this week, so the content provider 125 may choose toreduce the budget in order to reduce the number of impressions becausethe content provider 125 may not have sufficient inventory of flights.In some cases, the content provider 125 can choose to increase thebudget or the bid amount if the predicted performance is below athreshold (e.g., predicted number of in-application events is less thanthe current inventory). In some cases, the predicted performance can bewhat the content provider 125 desires, the content provider may notchange the input value. The data processing system 120 can configure,responsive to the indication of the predicted performance, theapplication content delivery profile of the content campaign for theapplication with the same or adjusted input value (e.g., increase ordecrease the bid amount or budget).

FIG. 2 is a flow diagram 200 illustrating operation of a system toreduce latency in feedback-based system performance determination inaccordance with an implementation. The system can include one or morecomponent of system 100, including, for example, computing device 110,data processing system 120, and application developer server 230 (e.g.,a server configured by application developer 125 to perform one or morefunction associated with the application). At 205, the data processingsystem 120 can provide a content item for an application. The dataprocessing system 120 can provide the content item in response to arequest for content for the computing device 110. At 210, the computingdevice 110 can transmit a request to the data processing system 120 todownload the application. At 215, the data processing system 120 canprovide the application to the computing device 110. Providing theapplication can include providing a data file comprising the applicationexecutable file or other software module, script, or code for theapplication. At 220, the computing device 110 can transmit an indicationof an in-application event. The in-application event can include, forexample, entering a search query in the application, achieving apredetermined score or level, or conducting a transaction in theapplication. The computing device 110 can be configured to transmit theindication of the in-application event directly to the data processingsystem 120. In some cases, the computing device 110 can be configured totransmit the indication of the in-application event to a server of theapplication developer 230, and the server 230 can forward or transmit aping to the data processing system 120 that indicates the in-applicationevent at 225.

The data processing system 120 can merge the ping data received at 225with internal data to generated merged data. The data processing system120 can apply a machine learning technique to the signals of the mergeddata to identify top ranking signals. The data processing system 120 canidentify a number of computing device 110 associated with the topranking signals to predict a performance of a content campaignconfigured to select content to provide to computing devices comprisingthe top ranking signals.

FIG. 3 is a graphical user interface (GUI) 300 of a system to reducelatency in feedback-based system performance determination in accordancewith an implementation. The GUI 300 can be generated or provided by oneor more component of system 100, including, e.g., data processing system120 or landscape generator component 140. The GUI 300 illustrates alandscape chart with an estimated or predicted performance. The GUI 300can include or refer to the application content delivery profile. Thelandscape chart can illustrate weekly installations versus a target costper installation (CPI); weekly in-application action volume versus atarget CPI (campaign-level selected conversions); or weeklyin-application action net value versus target CPI (campaign-levelselected conversions). The GUI 300 can allow the content provider 125 toinput one or more values for the application content delivery profile ofthe content campaign used to select content items or ads for theapplication.

The GUI 300 can allow the content provider 125 to optimize the contentcampaign for application installation volume 305 (e.g., the number ofinstallations of the application); installations from computing deviceslikely to complete in-application actions 310 (e.g., computing deviceslikely to conduct a transaction in the application); or installations oncomputing devices with users that are likely to spend the most 350(e.g., purchase the most flight tickets or most expensive flighttickets). The GUI 300 can further list different types of in-applicationactions or events, such as sign ups 325, first level completes 330,shares on social media 335, gem purchase 340, third game 345. Thecontent provider 125 can input the in-application actions. The dataprocessing system 120 can automatically determine the types of inapplication actions based on data associated with the application. Forexample, the data processing system 120 can obtain metadata for theapplication that lists the types of in-application events. Asillustrated here, the application contains 20 possible in-applicationactions 320. The content provider 125 can select one or morein-application events 360. As illustrated here, two in-applicationevents are selected: sign ups 365 and first level completes 370. Signsups 365 can refer to a user of the application signing up for an accountto access or use the application, signing up for a subscription to theapplication, or signing up for other content or services associated withthe application. The first level completes in-application action 370 cancorrespond to completing the first level in a game if the application isa game, or completing a first round of a multi-round process or task.

The data processing system 120 can receive an indication to optimize oneor more of 305, 310, or 355. The content provider 125 or applicationdeveloper 125 can select one or more 305, 310 or 355. If the contentprovider 125 selects to optimize installs from users likely to completein-application actions 310, the content provider 125 can further selectthe desired type of actions 360. Thus, the data processing system 120can receive, via GUI 300, a type of performance to optimize and a typeof in-application event to optimize.

The data processing system 120 can provide an input text box for atarget CPI 355. The content provider 125 or application developer 125can input, via GUI 300, the target CPI. For example, the target CPI canbe $3.50. The data processing system 120 can receive the target CPI viaGUI 300.

With the received one or more input values (e.g., 310, 360, and 355),the data processing system 120 can generate a predicted performance 380and 390. The predicted performance 380 comprises a graph having anX-axis with estimated CPI 385, and a Y-axis with estimatedinstallations. As shown in the graph 380, as the estimated CPIincreases, the estimated installations also increases. Marker 392corresponds to a target CPI of 3.50 as input at box 355. Marker 392indicates that at a target CPI of 3.50, the data processing system 120predicts or estimates there may be 1100 installations of theapplications. The 1100 installations of the applications may be causedby providing content item objects comprising a link or instruction todownload the application to at least 1100 computing devices 110.

The data processing system 120 can further estimate a number ofin-application actions as shown in chart 390. In chart 390, the x-axis385 corresponds to Estimated CPI and the Y-axis corresponds to estimatedor predicted in-application actions. As shown by marker 394, at a targetCPI of 3.50, the data processing system 120 estimates or predicts 3000in-application actions corresponding to sign ups 365 or first levelcompletes 370. These in-application actions may be attributed toproviding content items for the application for display on one or morecomputing devices. Configuring the application content delivery profilecan refer to or include setting the CPI via the input text box 355 orinput buttons 310 or 360.

FIG. 4 is an illustration of a method for reducing latency infeedback-based system performance determination in accordance with animplementation. The method 400 can be performed by one or more componentof system 100, such as a data processing system 120. In brief overview,at 405, the data processing system receives an indication of anin-application event and a first input value. At 410, the dataprocessing system receives a ping indicative of an occurrence of thein-application event. At 415, the data processing system merges datafrom the ping with internal data. At 420, the data processing systemprovides an indication of a determined predicted performance. At 425,the data processing system configures an application content deliveryprofile with a second input value.

At 405, the data processing system receives an indication of anin-application event and a first input value. The data processing systemcan receive the one or more indications from an application developerdevice or content provider device. The application developer device cancorrespond to a server or device accessed or used by an applicationdeveloper or content provider that provides or manages content items forthe application developer. The data processing system can receive anindication of an in-application event such as a sign up, first levelcomplete, search query with predetermined terms, sharing content of theapplication on a social media platform, purchasing content via theapplication, starting a new game, etc. The data processing system canreceive a first input value for the application content deliveryprofile. The first input value can include, for example, a target costper installation, optimization parameter (e.g., increase installations,increase installations for users likely to complete in-applicationactions, increase installations for users likely to spend the most), bidamounts, or budgets.

At 410, the data processing system receives a ping indicative of anoccurrence of the in-application event. The data processing system canreceive the ping via an interface from an application developed by theapplication developer and executed by a computing device remote from thedata processing system and different from the application developerdevice. The data processing system can receive the ping from theapplication developed by the application developer or a similarapplication. The ping can include ping data that indicates an occurrenceof an in-application event or an installation of the application on acomputing device.

At 415, the data processing system merges data from the ping withinternal data. The data processing system can merge data from the pingwith internal data determined by the data processing system to generatea merged data set. Internal data can refer to data stored at the dataprocessing system or otherwise obtained by the data processing system,including, e.g., application information, language, applicationconfiguration, etc. Merging the data can refer to associating ping datawith the internal data, storing the data in a common format or datastructure, using a common identifier for the ping data and the internaldata, or otherwise providing or translating the data to a merged dataset configured for machine learning.

In some cases, the data processing system can receive ping data from aplurality of applications. The plurality of applications can bedeveloped by the same or different application developers. The pluralityof applications can be in a same category as the application, have asame characteristic as the application, or be unrelated to theapplication. The data processing system can receive ping data indicatingoccurrences of one or more in-application events of the plurality ofapplications. The data processing system can categorize the plurality ofapplications to identify similar applications and similarin-applications events that correspond to the application and theselected in-application event. The data processing system can merge thesimilar applications and similar in-applications events with the pingdata and internal data to generate the merged data set used for machinelearning.

At 420, the data processing system provides an indication of adetermined predicted performance. The data processing system candetermine, based on the merged data set and the first input value, apredicted performance for the in-application event. The data processingsystem can provide, via a graphical user interface, an indication of thepredicted performance determined based on the merged data set and thefirst input value. In some cases, the data processing system candetermine the predicted performance for the in-application event basedon pings and internal data associated with the plurality of similarapplications.

At 425, the data processing system configures an application contentdelivery profile with a second input value. The data processing systemcan configure, responsive to the indication of the predictedperformance, the application content delivery profile with a secondinput value.

FIG. 5 is a block diagram of a computer system 500 in accordance with anillustrative implementation. The computer system or computing device 500can include or be used to implement the system 100, content provider125, application developer 125, computing device 110, content publisher115, online marketplace provider 175, data processing system 120,interface component 130, attribution component 135, landscape generatorcomponent 140, content selector component 145, and data repository 150.The computing system 500 includes a bus 505 or other communicationcomponent for communicating information and a processor 510 orprocessing circuit coupled to the bus 505 for processing information.The computing system 500 can also include one or more processors 510 orprocessing circuits coupled to the bus for processing information. Thecomputing system 500 also includes main memory 515, such as a randomaccess memory (RAM) or other dynamic storage device, coupled to the bus505 for storing information, and instructions to be executed by theprocessor 510. The main memory 515 can be or include the data repository155. The main memory 515 can also be used for storing positioninformation, temporary variables, or other intermediate informationduring execution of instructions by the processor 510. The computingsystem 500 may further include a read only memory (ROM) 520 or otherstatic storage device coupled to the bus 505 for storing staticinformation and instructions for the processor 510. A storage device525, such as a solid state device, magnetic disk or optical disk, iscoupled to the bus 505 for persistently storing information andinstructions. The storage device 525 can include or be part of the datarepository 155.

The computing system 500 may be coupled via the bus 505 to a display535, such as a liquid crystal display, or active matrix display, fordisplaying information to a user. An input device 530, such as akeyboard including alphanumeric and other keys, may be coupled to thebus 505 for communicating information and command selections to theprocessor 510. The input device 530 can include a touch screen display535. The input device 530 can also include a cursor control, such as amouse, a trackball, or cursor direction keys, for communicatingdirection information and command selections to the processor 510 andfor controlling cursor movement on the display 535. The display 535 canbe part of the computing device 110 or other component of FIG. 1, forexample.

The processes, systems and methods described herein can be implementedby the computing system 500 in response to the processor 510 executingan arrangement of instructions contained in main memory 515. Suchinstructions can be read into main memory 515 from anothercomputer-readable medium, such as the storage device 525. Execution ofthe arrangement of instructions contained in main memory 515 causes thecomputing system 500 to perform the illustrative processes describedherein. One or more processors in a multi-processing arrangement mayalso be employed to execute the instructions contained in main memory515. In alternative implementations, hard-wired circuitry may be used inplace of or in combination with software instructions to effectillustrative implementations. Thus, implementations are not limited toany specific combination of hardware circuitry and software.

Although an example computing system has been described in FIG. 5,implementations of the subject matter and the functional operationsdescribed in this specification can be implemented in other types ofdigital electronic circuitry, or in computer software, firmware, orhardware, including the structures disclosed in this specification andtheir structural equivalents, or in combinations of one or more of them.

Implementations of the subject matter and the operations described inthis specification can be implemented in digital electronic circuitry,or in computer software, firmware, or hardware, including the structuresdisclosed in this specification and their structural equivalents, or incombinations of one or more of them. The subject matter described inthis specification can be implemented as one or more computer programs,e.g., one or more circuits of computer program instructions, encoded onone or more computer storage media for execution by, or to control theoperation of, data processing apparatus. Alternatively or in addition,the program instructions can be encoded on an artificially generatedpropagated signal, e.g., a machine-generated electrical, optical, orelectromagnetic signal that is generated to encode information fortransmission to suitable receiver apparatus for execution by a dataprocessing apparatus. A computer storage medium can be, or be includedin, a computer-readable storage device, a computer-readable storagesubstrate, a random or serial access memory array or device, or acombination of one or more of them. Moreover, while a computer storagemedium is not a propagated signal, a computer storage medium can be asource or destination of computer program instructions encoded in anartificially generated propagated signal. The computer storage mediumcan also be, or be included in, one or more separate components or media(e.g., multiple CDs, disks, or other storage devices).

The operations described in this specification can be performed by adata processing apparatus on data stored on one or morecomputer-readable storage devices or received from other sources.

The term “data processing apparatus” or “computing device” encompassesvarious apparatuses, devices, and machines for processing data,including by way of example a programmable processor, a computer, asystem on a chip, or multiple ones, or combinations of the foregoing.The apparatus can include special purpose logic circuitry, e.g., an FPGA(field programmable gate array) or an ASIC (application specificintegrated circuit). The apparatus can also include, in addition tohardware, code that creates an execution environment for the computerprogram in question, e.g., code that constitutes processor firmware, aprotocol stack, a database management system, an operating system, across-platform runtime environment, a virtual machine, or a combinationof one or more of them. The apparatus and execution environment canrealize various different computing model infrastructures, such as webservices, distributed computing and grid computing infrastructures.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, declarative orprocedural languages, and it can be deployed in any form, including as astandalone program or as a circuit, component, subroutine, object, orother unit suitable for use in a computing environment. A computerprogram may, but need not, correspond to a file in a file system. Aprogram can be stored in a portion of a file that holds other programsor data (e.g., one or more scripts stored in a markup languagedocument), in a single file dedicated to the program in question, or inmultiple coordinated files (e.g., files that store one or more circuits,subprograms, or portions of code). A computer program can be deployed tobe executed on one computer or on multiple computers that are located atone site or distributed across multiple sites and interconnected by acommunication network.

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read only memory ora random access memory or both. The essential elements of a computer area processor for performing actions in accordance with instructions andone or more memory devices for storing instructions and data. Generally,a computer will also include, or be operatively coupled to receive datafrom or transfer data to, or both, one or more mass storage devices forstoring data, e.g., magnetic, magneto optical disks, or optical disks.However, a computer need not have such devices. Moreover, a computer canbe embedded in another device, e.g., a mobile telephone, a personaldigital assistant (PDA), a mobile audio or video player, a game console,a Global Positioning System (GPS) receiver, or a portable storage device(e.g., a universal serial bus (USB) flash drive), to name just a few.Devices suitable for storing computer program instructions and datainclude all forms of non-volatile memory, media and memory devices,including by way of example semiconductor memory devices, e.g., EPROM,EEPROM, and flash memory devices; magnetic disks, e.g., internal harddisks or removable disks; magneto optical disks; and CD ROM and DVD-ROMdisks. The processor and the memory can be supplemented by, orincorporated in, special purpose logic circuitry.

To provide for interaction with a user, implementations of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinventions or of what may be claimed, but rather as descriptions offeatures specific to particular implementations of particularinventions. Certain features described in this specification in thecontext of separate implementations can also be implemented incombination in a single implementation. Conversely, various featuresdescribed in the context of a single implementation can also beimplemented in multiple implementations separately or in any suitablesubcombination. Moreover, although features may be described above asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can in some cases be excisedfrom the combination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the implementations described above should not beunderstood as requiring such separation in all implementations, and itshould be understood that the described program components and systemscan generally be integrated in a single software product or packagedinto multiple software products.

References to “or” may be construed as inclusive so that any termsdescribed using “or” may indicate any of a single, more than one, andall of the described terms. References to at least one of a list ofelements can include any of a single element, more than one element, orall elements identified in the list.

Where technical features in the drawings, detailed description or anyclaim are followed by reference identifiers, the reference identifiershave been included to increase the intelligibility of the drawings,detailed description, or claims. Accordingly, neither the referenceidentifiers nor their absence have any limiting effect on the scope ofany claim elements.

Thus, particular implementations of the subject matter have beendescribed. Other implementations are within the scope of the followingclaims. In some cases, the actions recited in the claims can beperformed in a different order and still achieve desirable results. Inaddition, the processes depicted in the accompanying figures do notnecessarily require the particular order shown, or sequential order, toachieve desirable results. In certain implementations, multitasking andparallel processing may be advantageous.

1.-20. (canceled)
 21. A system to reduce latency for configurationupdates, comprising: a data processing system comprising one or moreprocessors and memory to: receive, from an application developer device,one or more indications of an in-application event comprising an audiointeraction and a first input value for an application content deliveryprofile; receive, via an interface from an application developed by anapplication developer and executed by a mobile computing device remotefrom the data processing system and different from the applicationdeveloper device, an indication of an occurrence of the in-applicationevent comprising the audio interaction on the mobile computing device;determine a number of installations of the application on other mobilecomputing devices; generate a first merged data set that mergesperformance signals for the in-application event across the other mobilecomputing devices on which the application is installed; identify aplurality of applications that are similar to the application based onat least one characteristic of the application; generate a second mergeddata set that merges performance signals for in-application eventscomprising audio interactions associated with installations of theplurality of applications; generate a model that combines the firstmerged data set with the second merged data set using a machine learningtechnique; determine, using the model, a predicted performance for thein-application event comprising the audio interaction based on the firstinput value; provide, via the interface, an indication of the predictedperformance determined based on the model and the first input value; andconfigure, responsive to the indication of the predicted performance,the application content delivery profile with a second input value,execution by the data processing system of the application contentdelivery profile configured with the second input value to cause thedata processing system to select content with an indication to installthe application.
 22. The system of claim 21, comprising: the dataprocessing system to identify the in-application event comprising theaudio interaction based on audio input detected by a sensor of themobile computing device.
 23. The system of claim 21, comprising: thedata processing system to determine the in-application event comprisingthe audio interaction based on speech detected by a sensor of the mobilecomputing device.
 24. The system of claim 21, comprising: the dataprocessing system comprising a digital assistant.
 25. The system ofclaim 21, wherein the mobile computing device comprises a digitalassistant.
 26. The system of claim 21, comprising: the data processingsystem to determine the predicted performance for the in-applicationevent comprising the audio interaction based on a regression model. 27.The system of claim 21, comprising: the data processing system todetermine the predicted performance for the in-application eventcomprising the audio interaction based on indications of occurrences ofin-application events associated with the plurality of applications. 28.The system of claim 21, comprising the data processing system to:determine a first predicted performance for the in-application eventcomprising the audio interaction based on a multivariate Poissonregression model; and calibrate the first predicted performance based ona second predicted performance of the plurality of applications todetermine the predicted performance.
 29. The system of claim 21,comprising the data processing system to generate the model with: datareceived from a first plurality of computing devices that installed theapplication; data received from a second plurality of computing devicesthat installed, responsive to a content item delivered by the dataprocessing system, one or more applications different from theapplication, the one or more applications matching a characteristic ofthe application; and data received from a third plurality of computingdevices that installed, responsive to an organic search, the one or moreapplications different from the application, the one or moreapplications matching the characteristic of the application.
 30. Thesystem of claim 21, comprising the data processing system to: provide asoftware development kit to the application developer device thatconfigured the application to transmit an indication to the dataprocessing system responsive to the occurrence of the in-applicationevent.
 31. A method of reducing latency for configuration updates,comprising: receiving, by a data processing system comprising one ormore processors and memory, from an application developer device, one ormore indications of an in-application event comprising an audiointeraction and a first input value for an application content deliveryprofile; receiving, by the data processing system via an interface froman application developed by an application developer and executed by amobile computing device remote from the data processing system anddifferent from the application developer device, an indication of anoccurrence of the in-application event comprising the audio interactionon the mobile computing device; determining, by the data processingsystem, a number of installations of the application on other mobilecomputing devices; generating, by the data processing system, a firstmerged data set that merges performance signals for the in-applicationevent across the other mobile computing devices on which the applicationis installed; identifying, by the data processing system, a plurality ofapplications that are similar to the application based on at least onecharacteristic of the application; generating, by the data processingsystem, a second merged data set that merges performance signals forin-application events comprising audio interactions associated withinstallations of the plurality of applications; generating, by the dataprocessing system, a model that combines the first merged data set withthe second merged data set using a machine learning technique;determining, by the data processing system using the model, a predictedperformance for the in-application event comprising the audiointeraction based on the first input value; providing, by the dataprocessing system via the interface, an indication of the predictedperformance determined based on the model and the first input value; andconfiguring, by the data processing system responsive to the indicationof the predicted performance, the application content delivery profilewith a second input value, execution by the data processing system ofthe application content delivery profile configured with the secondinput value to cause the data processing system to select content withan indication to install the application.
 32. The method of claim 31,comprising: identifying, by the data processing system, thein-application event comprising the audio interaction based on audioinput detected by a sensor of the mobile computing device.
 33. Themethod of claim 31, comprising: determining, by the data processingsystem, the in-application event comprising the audio interaction basedon speech detected by a sensor of the mobile computing device.
 34. Themethod of claim 31, wherein the data processing system comprises adigital assistant.
 35. The method of claim 31, wherein the mobilecomputing device comprises a digital assistant.
 36. The method of claim31, comprising: determining, by the data processing system, thepredicted performance for the in-application event comprising the audiointeraction based on a regression model.
 37. The method of claim 31,comprising: determining, by the data processing system, the predictedperformance for the in-application event comprising the audiointeraction based on indications of occurrences of in-application eventsassociated with the plurality of applications.
 38. The method of claim31, comprising: determining, by the data processing system, a firstpredicted performance for the in-application event comprising the audiointeraction based on a multivariate Poisson regression model; andcalibrating, by the data processing system, the first predictedperformance based on a second predicted performance of the plurality ofapplications to determine the predicted performance.
 39. The method ofclaim 31, comprising generating the model with: data received from afirst plurality of computing devices that installed the application;data received from a second plurality of computing devices thatinstalled, responsive to a content item delivered by the data processingsystem, one or more applications different from the application, the oneor more applications matching a characteristic of the application; anddata received from a third plurality of computing devices thatinstalled, responsive to an organic search, the one or more applicationsdifferent from the application, the one or more applications matchingthe characteristic of the application.
 40. The method of claim 31,comprising: providing, by the data processing system, a softwaredevelopment kit to the application developer device that configured theapplication to transmit an indication to the data processing systemresponsive to the occurrence of the in-application event.